Voice-controlled surgical system

ABSTRACT

Certain aspects of the present disclosure provide a voice-controlled system for controlling devices and systems in a surgical setting. The voice-controlled system includes at least one microphone and at least one loudspeaker distributed within the surgical setting to receive and audibly respond to voice commands from operating staff. In certain aspects, the at least one microphone and at least one loudspeaker include a phased microphone array and a phased loudspeaker array that are coordinated and synchronized to perform active noise reduction, echo cancellation, and directional determination of sound sources. The voice-controlled system performs tasks, such as activation of surgical devices, based on verbal interactions with the operating staff through the at least one microphone and at least one loudspeaker. In certain aspects, the voice-controlled system is configured to recognize, decipher, and prioritize natural language commands from different operating staff, and further provides pre-programming of device settings and operating staff pre-sets.

PRIORITY CLAIM

This application claims the benefit of priority of U.S. ProvisionalPatent Application Ser. No. 63/146,126 titled “VOICE-CONTROLLED SURGICALSYSTEM,” filed on Feb. 5, 2021, whose inventors are Steven T. Charlesand Paul R. Hallen, which is hereby incorporated by reference in itsentirety as though fully and completely set forth herein.

FIELD

The present disclosure generally relates to surgical devices andsystems, and more specifically, to voice-activated control systems forsurgical devices and systems.

BACKGROUND

Many surgical procedures, including ophthalmic procedures such asrefractive cataract surgery and vitreoretinal surgery, are extremelychallenging and require more than one operating staff member to adjustvarious apparatuses in the operating room, such as the lighting,operating table, microscope, viewing devices, and surgical tools and/orconsoles. The presence of multiple operating staff enables a surgeon tocontinue his or her work without the need to stop and change thesettings of a desired apparatus. However, simultaneous and seamlessoperation of separate devices or systems by multiple operating staff isa major challenge during surgical procedures, and in particular,ophthalmic procedures. Furthermore, extra personnel add cost to theprocedure and additional burden on resources within the operating room,such as floor space, while also increasing the risk of contaminationtherein.

Recently, voice-activated applications have been utilized to mitigatesome of the challenges of complex surgical procedures, thereby enablingvoice-control of surgical devices without the need for physicalinteraction by operating staff and reducing the amount of personnelrequired for surgical procedures. Yet, current voice-controlled surgicaldevices and systems have several limitations, such as the inability torecognize, discern, and prioritize between personnel providing voicecommands to control the devices. Additionally, in certain examples, thenumber and spatial arrangement of microphones within a high-noiseoperating room is suboptimal, leading to undetected or misinterpretedvoice commands from operating staff due to issues with background noiseand sound clarity. Still further, in certain examples, operating staffmust learn predetermined command input syntax to effectively executedesired device functions, rather than using conversational or naturallanguage commands.

Accordingly, there is a need in the art for improved voice-controlledsurgical systems.

SUMMARY

The present disclosure relates to surgical devices and systems, and moreparticularly, to voice-activated control systems for surgical devicesand systems.

According to certain embodiments, a surgical command system is provided.The surgical command system includes a processor, one or moremicrophones configured to convert sound waves within a surgicaloperating environment to one or more audio input signals relayed to theprocessor, one or more loudspeakers configured to generate sound waveswithin the surgical operating environment based on one or more audiooutput signals received directly or indirectly from the processor, and amemory comprising executable instructions in data communication with theprocessor. The processor is configured to execute the instructions tocause the surgical command system to directly or indirectly receive theone or more audio input signals from the one or more microphones,identify one or more speech commands in the one or more audio inputsignals, map at least one of the one or more speech commands to a userin the surgical operating environment, and identify one or more actionsassociated with the at least one of the one or more speech commands. Theprocessor is further configured to indicate the one or more actions to asurgical device to cause the surgical device to perform the one of moreactions, cause the one or more audio output signals to be produced basedon the one or more actions, and cause the one or more loudspeakers togenerate outgoing speech responses based on the one or more audio outputsignals.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above-recited features of the presentdisclosure can be understood in detail, a more particular description ofthe disclosure, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlyexemplary embodiments and are therefore not to be considered limiting ofits scope, and may admit to other equally effective embodiments.

FIG. 1 illustrates a surgical setting with a voice-controlled surgicalcommand system, in accordance with certain embodiments of the presentdisclosure.

FIG. 2 illustrates a schematic diagram of a voice-controlled surgicalcommand system, in accordance with certain embodiments of the presentdisclosure.

FIG. 3 illustrates exemplary components of the surgical command systemof FIG. 2, in accordance with certain embodiments of the presentdisclosure.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe Figures. It is contemplated that elements and features of oneembodiment may be beneficially incorporated in other embodiments withoutfurther recitation.

DETAILED DESCRIPTION

In the following description, details are set forth by way of example tofacilitate an understanding of the disclosed subject matter. It shouldbe apparent to a person of ordinary skill in the field, however, thatthe disclosed implementations are exemplary and not exhaustive of allpossible implementations. Thus, it should be understood that referenceto the described examples is not intended to limit the scope of thedisclosure. Any alterations and further modifications to the describeddevices, instruments, methods, and any further application of theprinciples of the present disclosure are fully contemplated as wouldnormally occur to one skilled in the art to which the disclosurerelates. In particular, it is fully contemplated that the features,components, and/or steps described with respect to one implementationmay be combined with the features, components, and/or steps describedwith respect to other implementations of the present disclosure.

Embodiments of the present disclosure generally relate to avoice-controlled system for controlling devices and systems in asurgical setting, such as an ophthalmic surgical setting. In certainaspects, the voice-controlled system includes one or more phasedmicrophone arrays (a phased microphone array refers to a plurality ofmicrophones arranged in a phased array) and one or more phasedloudspeaker arrays (a phased loudspeaker array refers to a plurality ofloudspeakers arranged in a phased array). In certain aspects, the one ormore phased microphone arrays are distributed throughout the surgicalsetting to identify and receive voice commands from operating staff.Further, one or more phased loudspeaker arrays may be distributedthroughout the surgical setting to output audio responses and otheraudible signals to the operating staff. In certain aspects, the one ormore phased microphone arrays are coordinated and synchronized toperform active noise reduction, echo cancellation, and directionaldetermination of sound sources. In certain aspects, the voice-controlledsystem performs tasks, such as activation of surgical devices, primarilybased on conversational or natural language interactions with theoperating staff through the one or more phased microphone andloudspeaker arrays. In certain aspects, the voice-controlled system isconfigured to decipher, learn, recognize, and prioritize verbal commandsfrom different operating staff, and further provides pre-programming ofdevice settings and operating staff pre-sets.

As used herein, the term “surgical setting” may refer to any environmentwhere surgical procedures are performed. For example, the term “surgicalsetting” may refer to an operating room, with one or more surgeons andoperating staff involved in the surgical setting.

As used herein, the term “surgical system” may refer to any surgicalsystem, console, or device for performing a surgical procedure. Forexample, the term “surgical system” may refer to a surgical tool orsystem, such as a phacoemulsification console, a laser system, animaging system, an intraocular lens (IOL) alignment system, a biometer,an optical coherence tomography (OCT) machine, or a vitrectomy console.

Although generally described with reference to an ophthalmic surgicalsetting, the devices and systems described herein may be implemented inother settings and contexts, such as other surgical settings, withoutdeparting from the scope of the present application.

As used herein, the term “about” may refer to a +/−10% variation fromthe nominal value. It is to be understood that such a variation can beincluded in any value provided herein.

FIG. 1 illustrates an example of a surgical setting 100 with avoice-controlled surgical command system 102, such as an ophthalmicsurgical setting, in which a surgeon 150, one or more additionaloperating staff, and a patient 112 are present, according to certainembodiments of the present disclosure. Although one surgeon 150 isillustrated in FIG. 1, multiple surgeons and/or operating staff may usethe surgical command system 102. For example, in certain embodiments, asurgical assistant and/or circulating nurse (i.e., circulator) may alsobe present within the surgical setting 100 and utilize the surgicalcommand system 102.

As shown, the surgical command system 102 includes a surgical commandcontroller 104 in direct or indirect communication with one or moresurgical systems, consoles, and/or devices (e.g., integrated in aninter-networked surgical suite) within the surgical setting 100, such asan operating table 120, a surgical console 122, a heads-up display 124,and a microscope system 126. Examples of suitable surgical systems thatmay be included in the surgical suite include surgical consoles forperforming vitreoretinal procedures, cataract surgeries, cornealtransplants, glaucoma surgeries, LASIK (Laser Assisted In SituKeratomileusis) surgeries, refractive lens exchanges, trabeculectomies,and refractive surgeries, among other consoles, imaging devices, laserdevices, diagnostic devices, and accessories identifiable by those ofordinary skill.

In certain embodiments, the surgical command controller 104 is astand-alone device or module (including a processor and a memory) thatis in wireless or wired communication with one or more surgical systemsphysically positioned within the surgical setting 100. In certain otherembodiments, however, the surgical command controller 104 includes oneor more processors and/or memory integrated within one or more of thesurgical systems physically positioned within the surgical setting 100.For example, the surgical command controller 104 may be integrated withthe surgical console 122, heads-up display 124, and/or microscope 126,as illustrated by phantom elements 104 in FIG. 1. In certain aspects,the surgical command controller 104 refers to a set of softwareinstructions that a processor associated with at least one of thesurgical systems, physically positioned within the surgical setting 100,is configured to execute. In certain aspects, the operations of thesurgical command controller 104 may be executed partly by the processorassociated with the surgical command controller 104 and partly in aprivate or public cloud.

The surgical command system 102 further includes one or more pluralitiesof microphones 106 arranged in phased arrays 136 and one or morepluralities of loudspeakers 108 arranged in phased arrays 138. Themicrophones 106 and loudspeakers 108 are in wireless or wiredcommunication with the surgical command controller 104, thus enablingthe controller 104 to receive voice commands provided by the surgeon 150and other operating staff, and further produce directional audibleresponses to the voice commands. In certain embodiments, the microphones106 and/or loudspeakers 108 are distributed within the surgical setting100 in close proximity to desired users (e.g., the surgeon 150, surgicalassistant, and/or circulating nurse) from which the voice commands areto be received. In certain embodiments, however, the microphones 106and/or loudspeakers 108 are widely dispersed within the surgical setting100 to provide greater coverage of the surgical setting 100.

Similar to the surgical command controller 104, the microphones 106and/or loudspeakers 108 may be stand-alone devices or may be physicallyintegrated with one or more other surgical systems within the surgicalsetting 100. For example, the microphones 106 and/or loudspeakers 108may be physically integrated into various components of the surgicalconsole 122, heads-up display 124, and/or microscope 126, as illustratedby phantom elements 106 and 108 in FIG. 1. In certain embodiments, oneor more phased microphone arrays 136 oriented toward (i.e., facing) oneor more users within the surgical setting 100 for directional listening,thus enabling focusing of desired sounds (i.e., voice commands) andsuppression of undesired sounds during capture and identification ofvoice commands in a noisy environment. For example, in certainembodiments, the microscope 126 may include at least one set of“forward-facing” microphones 106 for listening to the surgeon 150, andat least another set of “peripheral” microphones 106 oriented at 90° or180° relative to the forward-facing microphones 106 for listening to anoperating staff member (e.g., during a surgical procedure, the surgicalassistant may be located at positions 90° or 180° from the surgeon 150with respect to the microscope 126). In further embodiments, one or moresets of microphones 106 may be physically integrated into a surgicalface mask 130, headset, cap, lavalier, visor, glasses, or otheraccessory worn by the surgeon 150 or other operating staff during asurgical procedure. In such embodiments, the microphones 106 may bedisposable microphones, or hybrid reusable-and-disposable microphonesconfigured to be utilized with replaceable single-use filters.

The microphones 106 and/or loudspeakers 108 include any suitablemicrophones and/or loudspeakers arranged in a phased array to performbeamforming (e.g., beamsteering) in order to facilitate directionallistening, sound source localization and voice recognition, directionalaudio output (e.g., text-to-speech output), and audio signal qualityenhancement. For example, the microphones 106 perform receivebeamforming, or receive-side beamforming, while the loudspeakers 108perform transmit beamforming, or transmit-side beamforming. In certainembodiments, the beamforming microphones 106 arranged in phased arrays136 may enable the surgical command controller 104 to continuouslydetect a position of and localize a desired sound source (e.g., a userproviding voice commands) among many sources, and further capture andamplify the sound waves emitted by the source while reducing or ignoringbackground noise, reverberation, and feedback to enhance signal-to-noiseratio and voice recognition accuracy. For example, during a typicalophthalmic surgical procedure, the surgeon 150 sits at either a side ortop of the patient's head and near a microscope or other surgicaldevice, while the surgical assistant is near the side of the patient'shead and the circulating nurse moves around to several discretelocations within the surgical setting 100. In such an example, themicrophones 106 may detect a given voice command as originating from oneof the aforementioned positions, thereby enabling the surgical commandsystem 102 to identify the source of the voice command as either beingthe surgeon 150, surgical assistant, or circulating nurse.

Furthermore, the loudspeakers 108, in synchronization with themicrophones 106, enable the surgical command controller 104 to producedirectional audible responses, outputs, signals, and alerts to desiredusers within the surgical setting 100. For example, upon detection of avoice command and a position of either the surgeon 150, surgicalassistant, or circulating nurse generating the voice command, thesurgical command controller 104 may send a directionaltext-to-speech-type response targeted toward the aforementioned users asappropriate, which, in certain embodiments, may be preceded by a user'sname or other identifier. The directional response is output from theloudspeakers 108 and, in some examples, is directed via beamformingtowards the user who issued the voice command, thereby increasing thelikelihood that the commanding user is able hear the response moreclearly, e.g., in comparison to other persons in the room. A directionalresponse may also be advantageous because it may seem, to the commandinguser, as if the response originated from the system or device thecommanding user was addressing. The directionality of the responses andother outputs thus facilitates improved hearing thereof by operatingstaff within the surgical setting 100, and is also less likely to elicita response from or worry the patient 112.

In order to further enhance the quality of audio received by themicrophones 106, the microphones 106, loudspeakers 108, and surgicalcommand controller 104 are synchronized to perform active noisereduction (“ANR”) to eliminate or reduce continuous background noisesuch as noises produced by heating, ventilation, and cooling (HVAC)systems and surgical system and/or computer cooling fans. Additionally,the microphones 106, loudspeakers 108, and surgical command controller104 are configured to synchronously perform acoustic echo suppression orecho cancellation to more cleanly capture voice commands provided by thesurgeon 150 and/or operating staff. In certain embodiments, themicrophones 106 within the surgical setting 100 are tuned for highand/or low frequency sounds, and may be each utilized independently orin combination with other microphones 106. Similarly, the loudspeakers108 within the surgical setting 100 may each be utilized independentlyor in combination with other loudspeakers 108.

In certain embodiments, the microphones 106 and/or loudspeakers 108distributed within the surgical setting 100 may be utilized by thesurgeon 150 and/or other operating staff to listen to music and place orreceive phone calls. In such embodiments, the microphones 106 and/orloudspeakers 108 may connect to a user's (e.g., the surgeon's) ownmobile device via Bluetooth connection. When a phone call is placed orreceived by a user, the surgical command system 102 may give priority tothe phone call and automatically mute music playing through theloudspeakers 108.

In further embodiments, the surgical command system 102 includes one ormore microphones 106 and/or loudspeakers 108 directed toward andarranged in the vicinity of the patient 112, who is depicted as lying onthe operating table 120 in FIG. 1. For example, one or more microphones106 and/or loudspeakers 108 (individual devices or arranged in phasedarrays) may be disposed on or integrated with the operating table 120, apatient headrest, and/or under a drape 114 that covers the patient 112during a surgical procedure. The microphones 106 and/or loudspeakers 108may provide a communication channel between the patient 112 and anoperating staff member inside or outside the surgical setting 100, suchas the surgeon 150, surgical assistant, and/or circulating nurse, forimproving intelligibility during communication therebetween as well asaddressing hearing loss or removed patient hearing aids. For example,the microphones 106 and/or loudspeakers 108 may enable the surgeon 150to more clearly speak to the patient 112 in order to provide the patient112 with instructions and information, or calm the patient 112 duringthe surgical procedure. In certain embodiments, the loudspeakers 108 maybe utilized to provide soothing music to the patient 112, which furtherreduces the chance of the patient 112 listening to conversations betweenoperating staff members. In such embodiments, patient music may beprovided on separate channel from music provided to the surgeon 150and/or other operating staff, thus enabling the patient's music tocontinue while a phone call is placed or received through the othermicrophones 106 and/or loudspeakers 108 within the surgical setting 100.

Note that, although the microphones 106 and loudspeakers 108 aregenerally described above as being arranged in beamforming phasedarrays, individual directional microphones 106 and loudspeakers 108positioned near the surgeon 150, surgical assistant, circulating nurse,and/or patient 112 (e.g., under the drape 114) are also within the scopeof the disclosure.

As discussed in greater detail below with reference to FIG. 2 and FIG.3, the surgical command system 102 interfaces (e.g., wirelessly orwired) with the one or more devices and/or systems (e.g., surgicalconsole 122, heads-up display 124, and microscope 126) within thesurgical setting 100 to take one or more actions for operating thedevices and/or systems. The actions are based on voice commands providedby the surgeon 150 or other operating staff within the surgical setting100 that are received by the one or more phased microphone arrays 136distributed therein. In certain embodiments, the surgical command system102 can control (e.g., activate, deactivate, change operationalparameters, etc.) the various devices and/or systems within the surgicalsetting 100, adjust device settings, navigate a display dashboard(Graphical User Interface (GUI)/Human Machine Interface (HMI)), controldiagnostic devices, and provide alerts and/or recommendations to thesurgeon 150 and other operating staff via the phased array ofloudspeakers 108. With speech interaction, the surgical command system102 enables essentially “hands-free” surgical system control, thusimproving surgical procedure efficiency by reducing the amount ofmovement and/or “hands-on” device operation required of the surgeon 150and other operating staff, who may have their hands already occupiedwith other tasks.

FIG. 2 illustrates a schematic operational diagram 200 of thevoice-controlled surgical command system 102, according to certainembodiments of the present disclosure. As previously described, thesurgical command system 102 may be disposed within a surgical setting100, such as an operating room in which one or more users 210 (e.g.,surgeons 150 and/or operating staff) are present. During a surgicalprocedure, the user 210 generates a voice command 220 which, along withother sounds within the surgical setting 100 (e.g., ambient noise,speech from other users), is captured (e.g., picked up) by the phasedmicrophone arrays 136 of the surgical command system 102. The voicecommand 220 can be a simple phrase such as, but not limited to,“activate,” “deactivate,” “increase,” and “decrease,” or it can be acomplex phrase or sentence such as, but not limited to “increase 10%,”“decrease 10%,” “left 1 millimeter,” and “right 1 millimeter,” or evenmore complex phrases and/or sentences. In certain embodiments, the voicecommand 220 has a layered command architecture, wherein the user 210selects a desired system, a desired tool (e.g., device), a desired toolmode, and/or a desired task to be performed by the tool, for example,while in the tool mode. Each layer of the layered command architecture(e.g., system, tool, tool mode, and task) may correspond to a set ofinstructions (e.g., software instructions) that can cause thecorresponding surgical system to carry out one or more actions duringoperation. In still further embodiments, the voice command 220 is anatural language-type voice command 220, which is deciphered via anatural language processing (NLP) module of the surgical commandcontroller 104 with or without pre-programming, described in greaterdetail below.

Note that although the exemplary voice commands 220 described above arein the English language, the surgical command controller 104 may beconfigured to support any number of suitable languages, including butnot limited to English, Mandarin Chinese, Hindi, Spanish, French,Arabic, Portuguese, Russian, and the like.

As described above, the phased microphone arrays 136 receive the soundwaves of the voice command 220, as well as other sounds within thesurgical setting 100, and convert the sound waves into one or more audioinput signals 230 that are then directly or indirectly relayed to thesurgical command controller 104 of the surgical command system 102. Uponreceipt of the audio input signals 230, the surgical command controller104 identifies the voice command 220 in the audio input signals 230 viaa speech recognition module, identifies the source of the voice command220 via positional detection of the user 210 and/or a useridentification module, analyzes the voice command 220 via, for example,the NLP module, and maps the voice command 220 to the user 210 (e.g., auser profile) and a defined (e.g., user-defined) set of rules for theuser 210. The defined set of rules determine what instructions are to besent to a corresponding surgical system to carry out a desired actionindicated by the voice command 220. In certain embodiments, the definedset of rules may include user-defined actions to be executed uponcertain voice commands, as well as user-preferred system settings, toolmodes, tool sub-modes, task parameters, and the like. Since multipleusers may be present in the surgical setting 100 and use the surgicalcommand system 102 simultaneously, the surgical command controller 104is configured to identify (e.g., recognize) and distinguish voicecommands from each user within the surgical setting 100.

Voice identification is enabled in part due to directional listening bythe one or more beamforming, phased microphone arrays 136 distributedwithin the surgical setting 100, which facilitate sound sourcelocalization (i.e., positional detection) and suppression of unwantedoperating room noise, such as the speech of operating staff other thanthe surgeon 150 in certain instances. Additionally, the surgical commandcontroller 104 further includes a user identification module, describedin greater detail below with reference to FIG. 3, which works in tandemwith the NLP module and may be pre-programmed prior to a surgicalprocedure for voice recognition. The pre-programming of the useridentification module may include a brief, natural languageconversational sequence initiated between the surgical command system102 and one or more users within the surgical setting 100, wherein thesurgical command controller 104 learns speech patterns for each of theone or more users. During the surgical procedure thereafter, the useridentification module may perform a speech recognition algorithm onvoice commands 210 picked up by the phased microphone arrays 136 toidentify the source (e.g., user) of the voice commands 210 based on thelearned speech patterns.

The ability to identify the source of each voice command enables thesurgical command controller 104 to store and associate each user with apredetermined set of commands and/or rules, wherein each set of commandsand/or rules may correspond to a different set of instructions to becarried out by a corresponding apparatus. For example, after thesurgical command controller 104 analyzes and identifies the voicecommand 220 as being generated by a particular user 210, the surgicalcommand system controller 104 can map the voice command 220 to apredetermined set of rules for the user 210, which can include presetsystem settings, tool modes, tool sub-modes, task parameters, and thelike for each system and/or device within the surgical setting 100. Incertain embodiments, the predetermined set of rules for the user 210includes associations between simple phrases generated by the user 210and complex sets of predetermined instructions. For example, a simplephrase such as “display invert” may induce an inversion of heads-updisplay images with certain color presets preferred by the user 210. Thepredetermined set of rules for each user may be pre-programmed andstored into the surgical command controller 104 in tandem with the voicerecognition pre-programming sequence described above, wherein thesurgical command controller 104 learns speech patterns for each user.For example, during a tandem voice recognition and user presetprogramming sequence, the surgical command system 102 may first requesteach user to state their name and a device they want to address,followed by requests for desired system and/or device modes andnumerical parameters to be associated with corresponding voice commandsfrom the user.

The identification of the source of each voice command further enablesthe surgical command controller 104 to rank and prioritize voicecommands from certain users over others, which may be particularlybeneficial when multiple users generate voice commands simultaneously orwithin a short timeframe. Thus, the surgical command controller 104 maystore a predetermined hierarchy of users (e.g., user profiles) fromwhich voice commands may be received, and the predetermined hierarchy isutilized to prioritize voice commands from certain users over others.Alternatively, the surgical command controller 104 may suppress certainvoice commands from users determined to be a non-priority.

In tandem with performance of voice recognition, the surgical commandcontroller 104 analyzes the voice command 220 to determine the contentthereof and the intent of the user 210. In certain embodiments, analysisincludes matching of the voice command 220 to one or more commandspreviously pre-programmed by the user 210. However, because the surgicalcommand system 102 also supports natural language-type interactions withthe user 210, a more complex analysis may also be performed by the NLPmodule of the surgical command controller 104 to process and decipher(i.e., understand) complex natural language generated by the user 210.Thus, the surgical command system 102 facilitates use thereof withoutthe need for pre-programming or extensive syntax training by the user210.

After analyzing and mapping the voice command 220 to the user 210, thesurgical command controller 104 identifies one or more instructionsassociated with the voice command 220 and the user 210 that are based onthe predetermined set of rules for the user 210. The instructionsgenerally cause one or more actions to be taken or initiated by one ormore surgical systems within the surgical setting 100, such as theoperating table 120, surgical console 122, heads-up display 124, andmicroscope 126 depicted in FIG. 1. For example, in certain embodiments,the actions include system and/or device mode selection, system and/ordevice parameter selection or modification, system and/or deviceactivation and deactivation, actions for controlling operations ofsurgical tools, data transfer initiation, data recall, data entry,patient profile selection, surgical procedure parameter selection ormodification, video and photo control functions (e.g., record, pause,stop, snapshot), display control functions (e.g., image inversion, colorpreset selection), operative note dictation control functions (e.g.,record, stop, save, delete), phone control functions (e.g., answer orplace phone calls), and other procedural functions. For ophthalmicsurgical procedures, the actions may also include intraocular pressurecontrol via infusion system control, laser (e.g., retinal laser)parameter selection or modification, and the like.

As shown in FIG. 2, the surgical command controller 104 may optionallyproduce one or more audio output signals 240 based on the identificationor non-identification of actions associated with the voice command 220that are directly or indirectly relayed to the phased loudspeaker array138. The audio output signals 240 are converted by the plurality ofloudspeakers 108 into soundwaves forming an audible response 250 thatconfirms whether actions have or have not been identified by thesurgical command controller 104. For example, in certain embodiments,the loudspeakers 108 may generate a response 250 that includes a summaryof the identified actions (e.g., in simple or complex phrases), includesa simple recitation of the voice command 220, or includes a request forfurther information, clarification, or verification from the user 210,to which the user 210 may respond with another voice command. In certainembodiments, the response 250 may also include a request to the user 210to confirm or verify that the identified actions are correct. In stillfurther embodiments, the response 250 may indicate to the user 210 thatthe voice command 220 was not clearly received, and/or that an actionhas not been identified by the surgical command controller 104. Aspreviously described, the loudspeakers 108 include any suitablebeamforming loudspeakers that are arranged and configured to direct theresponse 250 to desired users within the surgical setting 100. Thus, thesoundwaves (i.e., sound wavefront) of the response 250 may be guideddirectly toward the user 210, thereby improving the ability for the user210, who generated the voice command, to hear the response 250 andconfirm, correct, clarify, or further supplement the voice command 250.In certain embodiments, the response 250 may also be directed bybeamforming so as to seem as if it is originating from the surgicalsystem to which the preceding voice command 220 was addressed.

In certain aspects, subsequent to providing the optional response 250,the surgical command controller 104 generates instructions 260 thatcorrespond to the identified tasks and are based on the predeterminedset of commands and/or rules for the user 210. The instructions 260 aredirectly or indirectly provided to one or more desired surgical systems270 associated with the identified tasks for execution thereof.Accordingly, the instructions 260 indicate to the appropriate surgicalsystems 270 the tasks identified by the surgical command controller 104and cause the surgical systems 270 to perform the identified actions,thereby fulfilling the objective of the voice command 220. Theinstructions 270 may be provided to separate surgical systems 270, or tosurgical systems 270 integrated into a single console 272, as shown inFIG. 2. In certain embodiments, a single set of instructions 270 maycause a plurality of surgical systems 270 to execute one or more actionssimultaneously or sequentially.

In certain embodiments, the surgical command system 102 furthercomprises a feedback mechanism that facilitates communication betweenone or more surgical systems within the surgical setting 100 and theuser 210 during or after performance of identified actions. For example,in certain embodiments, the surgical command controller 104 may producean audio output signal 280 that is converted by the plurality ofloudspeakers 108 into an audible response 290 that may convey to theuser 210 a warning alert, alarm, progress or status indicator, and anyother information relating to the performance of actions by the surgicalsystems within the surgical setting 100.

Through the inclusion of the components and systems disclosed herein, asurgical setting may be at least partially controlled by voice-activatedapplications with high confidence, thus providing hands-free operationof surgical systems during the performance of a surgical procedure. Thedisclosed embodiments allow a surgeon to control various functions ofsurgical systems without having to stop the surgical procedure to do so.Furthermore, by reducing the amount of physical interaction required tooperate surgical devices, the amount of personnel required for thesurgical procedure may be reduced, while also reducing the potentialrisk of bacterial and viral transmission (e.g., contamination) caused bypersonnel touching the surgical systems.

Embodiments of the present disclosure beneficially provide voice-controlof a surgical setting with the overall use of the system of which it isa part using a processor and memory within a controller of the surgicalsystem, as shown in FIG. 3, which illustrates exemplary components ofthe surgical command system of FIGS. 1-2, in accordance with certainembodiments

FIG. 3 illustrates an exemplary diagram showing how various componentsof the surgical command system 102 of FIGS. 1-2 communicate and operatetogether. As shown, the surgical command system 102 includes, withoutlimitation, the surgical command controller 104, microphones 106, andloudspeakers 108. The surgical command controller 104 includes aninterconnect 310, a network interface 312 for connection with a datacommunications network 350, and at least one I/O device interface 314,which allows for the connection of various I/O devices (e.g., themicrophones 106, loudspeakers 108, and surgical systems 270) to thesurgical command controller 104. The surgical command controller 104further includes a central processing unit (CPU) 316, a memory 318, andstorage 320. The CPU 316 may retrieve and store application dataresiding in the memory 318. The interconnect 310 transmits programminginstructions and application data, among the CPU 316, the networkinterface 312, the I/O device interface 314, the memory 318, and thestorage 320, etc. The CPU 316 can represent a single CPU, multiple CPUs,a single CPU having multiple processing cores, and the like.Additionally, the memory 318 represents random access memory.

The storage 320 may be a disk drive. Although shown as a single unit,the storage 320 may be a combination of fixed or removable storagedevices, such as fixed disc drives, removable memory cards or opticalstorage, network attached storage (NAS), or a storage area-network(SAN). Further, the storage 320 may comprise trained voice models 332 ofusers within a surgical setting, which includes user presets 334. Userpresets 334 comprise discrete sets of rules associated with each user ina surgical setting that are applied by the surgical command system 102to generate instructions to be carried out by a corresponding systemand/or device in response to a voice command by a user.

The memory 318 comprises a command module 322 that includesinstructions, which when executed by the processor, performs anoperation for controlling the surgical command system 102, as describedin the embodiments herein. For example, according to embodimentsdescribed herein, the memory 318 includes a speech recognition module324 which comprises executable instructions for recognizing (i.e.,identifying) speech, such as a voice command, in an audio input signalreceived from the microphones 106. In addition, the memory 318 includesa user identification module 326 having a voice model trainer 330, whichcomprises executable instructions for pre-programming of voice commands,learning speech patterns of users, and mapping the speech identified bythe speech recognition module 324 to a corresponding user. The memory318 further includes a natural language processing (NLP) module 328which comprises executable instructions for analyzing and decipheringnatural language voice commands (e.g., matching natural language totasks). Furthermore, the memory 318 includes a response module 332comprising executable instructions for generating audio output signalsbased on information received from the speech recognition module 324 toenable bi-direction communication between the surgical command system102 and a user.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover a, b, c,a-b, a-c, b-c, and a-b-c, as well as any combination with multiples ofthe same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b,b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

The foregoing description is provided to enable any person skilled inthe art to practice the various embodiments described herein. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments. Thus, the claims are not intended to belimited to the embodiments shown herein, but are to be accorded the fullscope consistent with the language of the claims.

Within a claim, reference to an element in the singular is not intendedto mean “one and only one” unless specifically so stated, but rather“one or more.” Unless specifically stated otherwise, the term “some”refers to one or more. All structural and functional equivalents to theelements of the various aspects described throughout this disclosurethat are known or later come to be known to those of ordinary skill inthe art are expressly incorporated herein by reference and are intendedto be encompassed by the claims. Moreover, nothing disclosed herein isintended to be dedicated to the public regardless of whether suchdisclosure is explicitly recited in the claims. No claim element is tobe construed under the provisions of 35 U.S.C. § 112(f) unless theelement is expressly recited using the phrase “means for” or, in thecase of a method claim, the element is recited using the phrase “stepfor.” The word “exemplary” is used herein to mean “serving as anexample, instance, or illustration.” Any aspect described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects.

What is claimed is:
 1. A surgical command system, comprising: one ormore microphones coupled to a processor and configured to convert soundwaves within a surgical operating environment to one or more audio inputsignals; one or more loudspeakers coupled to the processor andconfigured to generate sound waves within the surgical operatingenvironment based on one or more audio output signals received directlyor indirectly from the processor; a memory comprising executableinstructions; and the processor in data communication with the memoryand configured to execute the instructions to cause the surgical commandsystem to: receive directly or indirectly the one or more audio inputsignals from the one or more microphones; identify one or more speechcommands in the one or more audio input signals; map at least one of theone or more speech commands to a user in the surgical operatingenvironment; identify one or more actions associated with the at leastone of the one or more speech commands; indicate the one or more actionsto a surgical device to cause the surgical device to perform the one ormore actions; cause the one or more audio output signals to be producedbased on the one or more actions; and cause the one or more loudspeakersto generate outgoing speech responses based on the one or more audiooutput signals.
 2. The surgical command system of claim 1, wherein thesurgical command system is further configured to determine a location ofa source of the one or more speech commands in the one or more audioinput signal.
 3. The surgical command system of claim 2, wherein thelocation of the source is utilized to map the one or more speechcommands to the user.
 4. The surgical command system of claim 1, whereinthe surgical command system is further caused by the processor toactively reduce continuous ambient noise in the one or more audio inputsignals from the one or more microphones.
 5. The surgical command systemof claim 1, wherein the surgical command system is further caused by theprocessor to perform echo cancellation on the one or more audio inputsignals from the one or more microphones.
 6. The surgical command systemof claim 1, wherein the one or more actions comprise one or more of asurgical device selection, mode selection, and task selection.
 7. Thesurgical command system of claim 1, wherein the one or more actions areidentified at least partially based on a user profile associated withthe user and accessible to the processor.
 8. The surgical command systemof claim 7, wherein the user profile comprises mappings of differentactions to different speech commands, and wherein the mappings include amapping between one or more of the different actions and one or more ofthe different speech commands.
 9. The surgical command system of claim1, wherein: the processor being configured to map the at least one ofthe one or more speech commands to the user further comprises theprocessor being configured to map each of the one or more speechcommands to a corresponding user in a group of users including the user,the processor further configured to prioritize the at least one of theone or more speech commands and/or the one or more actions based on apredetermined hierarchy of the group of users, wherein the predeterminedhierarchy indicates a higher rank for the user compared to other usersassociated with the one or more speech commands.
 10. The surgicalcommand system of claim 1, wherein the outgoing speech responsecomprises an announcement of the one or more actions indicated to thesurgical device.
 11. The surgical command system of claim 10, whereinthe outgoing speech response is relayed from the one or moreloudspeakers in a direction of the user mapped to the identified one ormore speech commands.
 12. The surgical command system of claim 1,wherein the one or more loudspeakers include a plurality of loudspeakersarranged in a phased loudspeaker array.
 13. The surgical command systemof claim 12, wherein the phased loudspeaker array is arranged to performtransmit beamforming for transmitting soundwaves within the surgicaloperating environment.
 14. The surgical command system of claim 1, theone or more microphones include a plurality of microphones arranged in aphased array microphone array.
 15. The surgical command system of claim14, wherein the phased microphone array is arranged to perform receivebeamforming for receiving soundwaves within the surgical operatingenvironment.
 16. The surgical command system of claim 1, wherein the oneor more microphones are located in a surgical mask worn by the user inthe surgical operating environment.